*epistemic status: this is still being worked on and could have wrong info. i haven't done enough method evaluation to justify effort into making results*

[deep causal transcoding](https://www.lesswrong.com/posts/fSRg5qs9TPbNy3sm5/deep-causal-transcoding-a-framework-for-mechanistically) could actually be fairly compatible with circuits in superposition, and more recently, work on handcoding mlps to [implement fact memorization](https://www.lesswrong.com/posts/KWtchKwwnJkd4bwCi/challenge-hand-coding-weights-for-efficient-sequence-1)

more granular than the fact that dcts elicit network generalizations, is that they can elicit shallow feature interactions which cause some kind of “fact readoff” operation to happen in the net. downweighting refusal also seems fairly compatible with editing some shallow fact about the net at some intermediate state, if not perhaps a little more computational than merely reading off facts. my best guess is that refusal training instills a similar factual association in the net, which could be why it is so easy to steer/train away refusal

in a sense this seems plainly obviously true. an addition circuit under average LLM prior does way more algorithmic and deep stuff than fact-related stuff should involve. but fact stuff could also comprise a lot of the net, and circuits might not even compose in weird convoluted ways

<!--
in other news, I take [john wentworth](https://www.lesswrong.com/posts/tEPHGZAb63dfq2v8n/how-useful-is-mechanistic-interpretability?commentId=CcAHnxGNAHCJnLy29) literally and think about ensembles of shallow circuits as markov blankets, because this could have cool mathematical properties. I then notice markov blankets are circle shaped and in neural nets, computationally isolated nodes aren't geometrically clustered. also it's unclear my method even works to the reliability that markov blanketing makes sense (certainly not 99% reliability). could be wrong about this, but unrelated
-->

about the current thing: optimizing [this term](https://www.lesswrong.com/posts/8iG7orf6q8hNHT4Xh/ajskateboarder-s-shortform?commentId=vyDRwt434Y9pz9Jgm) (technically the term is halved but whatever) gives you shallow feature interactions from bilinear mlp weights. on swiglu it also performs pretty well for activations. im not too sure what all to try with it yet. ~~I'm probably gonna train a bilinear python predicting transformer and work on that (even if python is very structured the transformer will probably not entirely implement deep circuitry)~~. this may even be optimal for the bilinear mlps, because $Lx \odot Rx$ is an exactly second-order taylor expansion, and swiglu is potentially bounded on this too, despite Swish being slightly more expressive. so circuits could naturally be second order feature interactions[^1]

obviously the current term $\frac{1}{2}(u^{\top}H_x[l,r])^2$ does not guarantee no intermediate features also interact, so you inherently have lesser mechanistic understanding, and this also doesn't prove shallow circuits per se. not that existing work was much evidence anyways. most work supposes we are skipping deep computation with shallow representations, and other work, e.g. dcts are ok with this tradeoff because they only really care about eliciting stuff. so the whole strategy is maybe wrong since, now we want to figure out and minimize all the other potential feature interactions. so back to pure sparsity/reconstruction?

sensible counterarg: shallow stuff is nonetheless smeared across a noticably large slice of layers (and this is somehow possible to test). I guess the sensible thing to do would be to get some kind of privilegedness measure on the circuit as you sample from more midpoints of your transformer slice. hope that bilinear weight properties would make this easier to test

### adding bilinear unit participation ("sparsity")

alternatively, you can optimize against more features interacting instead of including more interactions. you can rely on a less problematic sparsity measurement (participation ratio). if you want to minimize the effective distribution of computation with the effect of some two non-identical features on some output feature (i.e. you want to find actually shallow circuits), suppose $\Delta^{s \rightarrow t}(x+\theta)\text{ where }\theta = \alpha l+\beta r$ is what's being modeled. perturb at layer $s$ w/ $\theta$, for the exclusive slice $M_{s+1, \dots, t-1}$ compute the mixed Hessian wrt $\theta_{1,2}$ over the MLP activations before the down projection[^2]:

$$\begin{aligned}
c_{\ell,i}(x)=
\left.
\frac{\partial^2}
{\partial \theta_1\partial\theta_2}
\phi_\ell\!\left(
M_{s,\dots,\ell}
(x+\theta_1+\theta_2)
\right)
\right|_{\theta_1=\theta_2=0} \\

c_i(x)=\operatorname{concat}\left[c_{s+1,i}(x),\ldots,c_{t-1,i}(x)\right] \in \mathbb{R}^{D}
\end{aligned}
$$

where $\phi$ is just the MLP forward prop stuff that happens up to the down projection, and $D = d_\mathrm{MLP}\times(t-s-1)$. in practice $c_i(x)$ gives a numerical evaluation of the mixed Hessian for every hidden unit in the slice at the origin. then measure the participation ratio for every hidden unit:

$$\begin{aligned}
\mathrm{PR}_{x,i}=
\frac{\left(\sum_{j=1}^{D}c_{x,i,j}^2\right)^2}
{\sum_{j=1}^{D}c_{x,i,j}^4} \\
\overline{\mathrm{PR}}_{x,i}=
\frac{\mathrm{PR}_{x,i}}{D}
\end{aligned}
$$

then use it as a penalty for the existing term:

$$\underbrace{\frac{1}{2}(u_i^{\top}H_x[l_i,r_i])^2}_{\text{causal importance}}-\underbrace{\lambda S \overline{\mathrm{PR}}_{x,i}}_{\text{unit participation}}$$

$\lambda$ is yet another regularization hyperparameter (0.1 seems to be ok), and S is a parameter to weigh PR updates against per-factor "causal importance," calculated by fitting a DCT only using the "causal importance" term on the same datapoints, and finding:

$$S=\frac{1}{N^2}\sum_x\sum_i (u_i^{\top}H_x[l_i,r_i])^2$$

for transformers using bilinear layers, this seems to give you weight circuits that are sparse in the neuron basis and connect across a wide layer range. the tail ends of causal importance for every circuit tend to grow wider, suggesting not all circuits are naturally structured this way while some very much are. you can evaluate how causally important/sparse circuits are by just using the objective function on circuits with different inputs; more concrete, you can just run these metrics on datapoints to see how causally strong/sparse the features interactions are:

$$\frac{1}{2}(u_i^{\top}H_x[l_i,r_i])^2 \qquad \overline{\mathrm{PR}}_{x,i}$$

and overall it seems that you can reduce average participation ratio from ~300-400 to ~1 (making circuits ~400x more sparse), and many circuits will retain 70-100% of their causal strength on the target feature, or even increase in some cases. a few only retain 50% of their strength which may point to more inherently distributed computation

*(this was wrong)*

maximizing the PR term instead of minimizing would probably give you more of the same circuits in existing DCTs, that is, most of their computation is distributed across many different neurons that you couldn't hope to evaluate easily (though i do wonder how else you can modify the "unit participation" term to do interesting stuff)

### issue: pure Hessian symmetry

probably should've been able to catch this earlier, but $H_x[l_i,r_i]$ is [symmetric](https://en.wikipedia.org/wiki/Symmetry_of_second_derivatives) to $H_x[r_i,l_i]$, so nothing in principle is stopping the $l,r$ features (henceforth, lr-features) from collapsing into directionally similar factors (as what turned out to happen). this was because i was not using the bilinear layer the way i wanted to use it. you should have $\Delta^{s \rightarrow t}(x, \alpha l, \beta r)$ inject $L = \alpha l, R = \beta r$ at the source bilinear $Lx \odot Rx$, and use the partial mixed Hessian wrt the lr-features instead of wrt $\theta = \alpha l + \beta r$. of course, this is now specific to interpreting bilinear-layer transformers, and not starting from the residual stream. but it seems to work a lot better, and by default i've observed pretty good unit sparsity in the source and target layers. adding back the PR term to reduce MLP unit interactions to ~1 per layer gave ~useless circuits, which is definitely more like what i expected

### further improvements

the default approach still largely doesn't explain attention routing. this bugged me enough that i tried reapplying the PR term over attention. instead of concatenating the feedforward net activations and taking the (problematic) pure Hessian over feedforward MLP unit contributions (which bilinear layers don't have, and it anyways doesn't help now), concat the [OV circuit](https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J#z=CLmGoD1pvjmsg0dPyL3wkuGS) contributions over the span of layers, corresponding to attn head contributions[^3]. the PR measure is otherwise calculated the same, where $D = d_\mathrm{model}\times(t-s)$ i think. on this note it also doesn't make sense to exclude layers s and t (this should've been done anyways). $\lambda = 0.03$ seems like a good choice for regularizing the penalty

so far results here seem more useful; i.e. stronger feature interactions, reduced distribution computation across OV circuits. and significantly more stable feature convergence[^4]. this probably refutes my concerns about the method now selecting significantly different circuits than without the PR term

### what do the circuits look like

they're not circuits in the sense that they fully explain some computation/algorithm to the unembed of the model, but they are somewhat circuits? some set of hidden units at the source layer activate a variety of attention heads (OV circuits) across layers, some far more strongly than others, and this leads to the downstream U feature activating. i'm unsure how easy it would be to explain the downstream contribution to the U feature from attention heads (assuming squared attention doesn't work, probably quite difficult). but this is getting somewhere in terms of mechanistically explaining things, at least compared to having no way to mechanistically explain the feature connections

some representative-ish examples:

![250](<./Pasted image 20260926180802.png>) ![250](<./Pasted image 20260926180939.png>) ![250](<./Pasted image 20260926181212.png>)

and some of these factors also look interpretable. i don't think these factors are comparable to SAE atoms (mainly because I would expect SAEs to encourage more fine-grained separation of data features), but it is still pretty interesting. here's a metric i got for locating these[^5]

![250](<Pasted image 20260925221314.png>)

the x-axis basically measures the magnitude of vocabulary shift after adding the lr-features as MLP biases, and y-axis measures how reliably this vocabulary shift happens across random samples. the top-right region contains some interesting factors (somewhat cherry-picking from this region):

- ![200](<./Pasted image 20260925222438.png>)
  *(political terms vs. professional terms)*
- ![200](<./Pasted image 20260925222728.png>)
  *(uppercase vs. lowercase)*
- ![200](<Pasted image 20260925222955.png>)
  *(singular vs. plural)*

when steering on unit lr-features, this token shift doesn't seem to strongly reverse or attenuate to insignificance (of course, there will be some attenuation if you are training on a early/mid-layer slice). keep in mind these factors were learned using 32 samples of specifically jailbreak text, over multiple averaged tokens, across layers. seems pretty neat

### other thoughts

- this whole approach initially felt unnatural for discovering things true to the model, as MELBO in general is mostly framed around eliciting things, and it vaguely sounds like you're doing pathological perturbations on the model in order to get certain behaviors; i don't think this is necessarily true though. [this post](https://www.lesswrong.com/posts/SNAKJuN8FdoEaWeFC/in-search-of-natural-features#The_Amp_function__math) and [shortform](https://www.lesswrong.com/posts/ZBjhp6zwfE8o8yfni/stefanhex-s-shortform?commentId=NozNwZPZHBW2Ry2PF) have also made this direction more convincing to me, and i think it would be really cool to see more MELBO-flavored approaches take shape
- it quickly grows in VRAM allocation (learning 1 factor needs ~300MiB), but this could also be largely downstream of the quadratic DCT

[^1]: not that this is an extremely distinctive structure per se

[^2]: in general, $\theta_{1,2}=\alpha l,\beta r$ respectively

[^3]: they're contributing whatever it is that OV circuits contribute

[^4]: the DCT is initialized from random vectors

[^5]: mostly via schlepping with the gpt